PATENT 

WFVA/NOKIA File Nos . 944-003-069/32287 



UNITED STATES PATENT APPLICATION 

of 

Olli Aaltonen, 
Satu Jaaskelainen, 
Seppo Karjalainen, 

Heikki Lang, 
Heikki Hama.la.inen, 
Stina Ojala, and 
Terho Kaikuranta 
for a 
MOBILE PHONE 
FEATURING AUDIO -MODULATED VIBROTACTILE MODULE 



PATENT 

WFVA/NOKIA File Nos . 944-003-069/322 87 

MOBILE PHONE 
FEATURING AUDIO -MODULATED VIBROTACTILE MODULE 

BACKGROUND OF THE INVENTION 

1. Field Of Invention 

5 The present invention relates to a 

telecommunications network; and, more particularly, to a 

mobile phone used therein. 



2. Description of Related Art 

Many people are either hard of hearing or cannot 

10 hear at all. These people have a very hard time using a 

mobile phone, or cannot use it at all. 

In the prior art, there are only a few products that 
address this problem, including a hearing aid or a mobile 
phone loop headset. However, the hearing aids normally 

15 cannot be used with mobile phones. The NMP loop headset 

is one solution, but this requires that a significant 
amount of hearing capability still exists, and only the 
user's ears are used as an input channel. 

Moreover, vibrotactile representation of speech 

20 stimuli is known in the art. In medical science, the 

vibrotactile-phenomenon is commonly practiced in 
audiology. See "Vibrotactile Perception of Speech, " by 
Stina Ojala, Phonetics, University of Turku, Finland, for 
a full explanation of the vibrotactile phenomenon itself. 

25 Many different vibrotactile device are known, including 

devices mounted on an ear mould, a ring, a bracelet or a 
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watch. These devices convert speech into a vibration 
modulation that stimulates and assists the user. 

SUMMARY OF INVENTION 
The present invention provides both a method and 
apparatus for assisting and improving hearing capability 
by feeding mechanical vibration to the fingers or other 
suitable location. This vibration is modulated by a 
received speech signal. This invention provides a way of 
applying the vibrot ac t i le - phenomenon to use in mobile 
phones and a manner of implementing the same. The 
benefit of having this feature is to make hearing easier 
in case of reduced hearing capability or very loud 
background noise. 

In its broadest sense, the present invention 
features a telecommunications network having a mobile 
phone with an audio-modulated vibrotactile module that 
responds to a telecommunications signal containing 
information about incoming speech from a called/calling 
party, for providing an audio-modulated vibrotactile 
module force containing information about the incoming 
speech from the called/calling party to vibrate a user's 
fingers, facial skin, wrist, cheek or other suitable 
location. 

In one embodiment, the audio-modulated vibrotactile 
module comprises an audio- to-vibrotactile converter that 
responds to the telecommunications signal, for providing 
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an audio-to-vibrotactile converter signal containing 
information about a vibration modulation of the incoming 
speech from the called/calling party. The audio- 
modulated vibrotactile module also has a vibrotactile 
5 actuator that responds to the audio-to-vibrotactile 

converter signal, for providing the audio-modulated 
vibrotactile module force in the form of a vibrotactile 
actuator force . 

In another embodiment, the telecommunications system 
=5 10 would comprise the audio-to-vibrotactile converter, and 

the audio-modulated vibrotactile module in the mobile 
phone would contain only the vibrotactile actuator. 

The vibrotactile actuator may be an 
electromechanical actuator arranged in the housing of the 
15 mobile phone for providing vibration to a user's fingers, 

wrist or facial skin. 

The present invention also includes a method 
comprising two steps, i.e. a converting step and a 
transforming step. First, a telecommunications signal 
2 0 containing information about incoming speech from a 

called/calling party is converted into an audio-to- 
vibrotactile converter signal containing information 
about a vibration modulation of the incoming speech from 
the called/calling party. Next, the audio-to- 
25 vibrotactile converter signal is transformed into an 

audio -modulated vibrotactile module force containing 
information about the incoming speech from the 
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called/calling party to vibrate a user's fingers, facial 
skin, wrist, cheek or other suitable location. 

The step of converting may include performing 
vibration modulation using frequency domain filtering or 
5 equalization, linear/non- linear amplification, mixing 

speech signals, or a speech encoding algorithm. 

The step of transforming may include 
electromechanically or acoustically actuating the audio- 
modulated vibrotactile module force. 

10 Implementation in a Mobile Phone 

In particular, in the mobile phone an incoming voice 
information is used as a source for vibration modulation. 
Speech is fed directly or it can be modified by one or 
more known methods before it is sent to the transducer 

15 component that transforms electrical signals to 

mechanical vibration. The later modification can be done 
by hardware or by software, and it can perform a specific 
algorithm if so specified. 

If a hardware approach is used, a relatively simple 

20 modification may be performed, such as frequency domain 

filtering or equalization, linear/non- linear 
amplification or mixing speech signal with other 
signal (s) . A more complex modification may add 
considerable size, cost and component number within a 

2 5 device. 
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If a software approach, is used, typically a 
calculation method or an algorithm can be used to modify 
the incoming speech signal. By using this signal as an 
input signal, a new output signal is determined according 
5 to the algorithm so that optimal vibration will be 

produced. An advantage of the software approach is that 
a known speech encoding algorithm can be utilized within 
the mobile phone to produce input parameters for the 
actual modification purpose for vibrotactile 

10 functionality. In the best case, only minor additions to 

the existing speech coding software would need to be 
made, and therefore the processing capacity requirement 
will not be significant, which makes this feature quite 
easy to include in existing phones from the software 

15 point of view. Another advantage is that, if needed, the 

vibration defining parameters can be adjusted by the user 
through a user interface, where the user may be the 
person him/herself, the doctor of the person, or where 
parameters such as either direct numerical parameters or 

20 a preset-list to select from may be used. 

The electromechanical actuator may be any suitable 
component that is able to produce feasible vibration 
located within the product so that the optimal effect can 
be introduced. Vibration control signal is fed into this 

25 component, and a vibration is felt on the fingers, facial 

skin, wrist, cheek or other suitable locations. 
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The so-called F0 -parameter (phonetic term, 
fundamental frequency) of the speech may be used for 
coding (signal modification) , which helps one to 
interpret the incoming speech in a noisy environment. 
5 This makes it is possible to take into account the 

intonation and phonetic weight of the speech, which is an 
important property in some languages, such as Chinese, 
Vietnamese, etc., in which the tone and height of the 
voice is very important. 
10 An acoustical transducer made to fit inside the 

user's ear could incorporate the vibrotactile feature or 
functionality, for example, such a transducer could 
easily be applied via the phone itself or another small 
device, such as audio transducer or accessory device. 

15 Implementation in a Network 

The audio-to-vibrotactile conversion processing may, 
in some cases, require so much processing power that it 
may be more feasible to implement this functionality in 
the telecommunications network. If a network-based 

20 solution is used, this feature will work with older 

phones as well and more processing power can be used. 
Also, a separately-priced network service can be provided 
to a specific customer. In this case, a phone subscriber 
will make a personal agreement with a network operator to 

25 use this kind of service, which is then utilized as 

associated with the specific telephone number in 
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question. Network service can also utilize personalized 
hearing parameters adjusting the speech processing. 
These personal parameter (s) can be either selected by 
trial-error based, present values or personalized values 
5 given by a customer's physician. 

BRIEF DESCRIPTION OF THE DRAWING 
The drawing includes the following Figures: 
Figure 1 is a diagram of one embodiment of a mobile 
phone in a telecommunications network that is the subject 
10 matter of the present invention. 

Figure 2 is a diagram of a vibrotactile module shown 
in Figure 1 that is the subject matter of the present 
invention. 

Figure 3 is a diagram of a hardware -based signal 
15 processor that is part of an audio-to-vibrotact ile 

converter shown in Figure 2 . 

Figure 4 is a diagram of a frequency domain 
filtering or equalization module that forms part of the 
hardware -based signal processor shown in Figure 3. 
2 0 Figure 5 is a diagram of a linear/non- linear 

amplification module that forms part of the hardware- 
based signal processor shown in Figure 3 . 

Figure 6 is a diagram of a speech mixing module that 
forms part of the hardware -based signal processor shown 
25 in Figure 3 . 
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Figure 7 is a diagram of a software-based signal 

processor that forms part of an audio- to-vibrotactile 

converter shown in Figure 2 . 

Figure 8 is a diagram of a speech encoding algorithm 
5 module that forms part of the software-based signal 

processor shown in Figure 7 . 

Figure 9 is a diagram of an electro-mechanical 

actuator that forms part of a vibrotactile actuator shown 

in Figure 2 . 

10 Figure 10 is a diagram of an acoustic actuator that 

forms part of a vibrotactile actuator shown in Figure 2. 
Figure 11 is a diagram of another embodiment of a 

mobile phone in a telecommunications network that is the 

subject matter of the present invention. 
15 Figure 12 is a diagram of a personalized hearing 

parameters module of an audio- to-vibrotactile converter 

shown in Figure 11. 

DETAILED DESCRIPTION OF INVENTION 
Figure 1: Mobile Phone 
2 0 Figure 1 shows a mobile phone generally indicated as 

10 for use in connection with a telecommunication network 
generally indicated as 12 . The scope of the invention is 
also intended to cover other user equipment and mobile 
electronic devices, such as a portable computer. 
25 The mobile phone 10 includes a signal processor 12 

connected to a radio access network module 14 having an 
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antenna 16, a display module 18, an audio module 20, a 
microphone 22, a read only memory 24 (ROM or EPROM) , a 
keyboard module 2 6 and a random access memory 2 8 (RAM) , 
which, are all known in the art. The operation of the 
5 signal processor 12 in relation to the aforementioned 

elements is also known in the art. Moreover, the scope 
of the invention is not intended to be limited to any 
particular kind or type of these elements. For example, 
the scope of the invention is intended to include the 
10 radio access network module 14 being an antenna module 

14, a radio frequency (RF) module, a radio modem or the 
like. 

The signal processor 12 is also connected to an 
audio-modulated vibrotactile module 30. The whole thrust 
15 of the invention relates to the operation of the audio- 

modulated vibrotactile module 30 with respect to the 
voice modulation of incoming speech into a vibrotactile 
sensation that can be sensed by the user of the mobile 
phone 10. 

20 In operation, the audio-modulated vibrotactile 

module 3 0 responds to a telecommunications signal 
containing information about incoming speech from a 
called/calling party 11, and provides an audio-modulated 
vibrotactile module force containing information about 

25 the incoming speech from the called/calling party 11 to 

vibrate a user's fingers, facial skin, wrist, cheek or 
other suitable location. 
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As discussed below, the audio- to- vibrotactile 
conversion may be performed in the mobile phone 10 as 
shown in Figure 1 or in the telecommunications network as 
shown and described below in relation to Figure 11. 



5 Figure 2: Audio-Modulated Vibrotactile Module 

Figure 2 shows in detail the audio-modulated 
vibrotactile module 3 0 having an audio-to-vibrotactile 
converter 40 and a vibrotactile actuator 42. 

In operation, the audio-to-vibrotactile converter 40 
10 responds to the aforementioned telecommunications signal, 

ji and provides an audio-to-vibrotactile converter signal 

containing information about a vibration modulation of 
the incoming speech from the called/calling party 11. 

The vibrotactile actuator 42 responds to the audio- 
15 to-vibrotactile converter signal, and provides the audio- 

modulated vibrotactile module force in the form of a 
vibrotactile actuator force that contains the information 
about the incoming speech from the called/calling party 
11 to vibrate a user's fingers, facial skin, wrist, cheek 
20 or other suitable location. 

The audio-to-vibrotactile converter 40 may be 
implemented in hardware or software, as discussed below. 
The scope of the invention is not intended to be limited 
to any particular implementation thereof. 
25 The vibrotactile actuator 42 may be an 

electromechanical actuator 70 (Figure 9) arranged in a 
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housing generally indicated in Figure 1 as 10a of the 
mobile phone 10 (Figure 1) for providing a vibration to a 
user's fingers, wrist or facial skin. Vibrotactile 
actuators are known in the art, and the scope of the 
5 invention is not intended to be limited to any particular 

type or kind thereof. 

Figures 3-6: Hardware -Based Vibration Modulation 
Figure 3 shows a hardware -based signal processor 5 0 
for performing a vibration modulation in the audio-to- 

10 vibrotactile converter 40. In the hardware -based signal 

processor 50, the vibration modulation may be performed 
using a frequency domain filtering or equalization module 
52 shown in Figure 4, a linear/non-linear amplification 
module 54 shown in Figure 5, or a speech mixing module 56 

15 shown in Figure 6 for mixing speech signals with other 

signals . 

The hardware -based signal processor 50 may be 
implemented to perform any one of the aforementioned 
functions using a microcontroller-based design. As a 

20 person skilled in the art would appreciate, the 

microcontroller-based design would typically include an 
inexpensive microcontroller, ROM, RAM, input/output 
devices and data and address lines for coupling the same. 
The scope of the invention is not intended to be limited 

25 to any particular hardware implementation of the signal 

processor 50. 
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Figures 7-8: Software -Based Vibration Modulation 
Figure 7 shows a software-based signal processor 60 
for performing vibration modulation in the audio-to- 
vibrotactile converter 40. 
5 In the software-based signal processor 60, the 

vibration modulation may be performed using a speech 
encoding algorithm module 62 shown in Figure 8. 

The software-based signal processor 6 0 may be 
implemented to perform any one of the speech encoded 
10 algorithm using a microprocessor-based design. As a 

person skilled in the art would appreciate, the 
microprocessor-based design would typically include a 
more expensive processor, ROM, RAM, input/output and data 
and address lines for coupling the same. The scope of 
15 the invention is not intended to be limited to any 

particular software implementation of the signal 
processor 60. 

Moreover, the software-based signal processor 60 may 
respond to user adjustable vibration defining parameters 
20 received from the user of the mobile phone 10 (Figure 1) , 

including direct numerical parameters or a pre-set list 
of parameters, as discussed above. 
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Figures 9-10: Vibrotactile Actuator 
The vibrotactile actuator 42 (Figure 2) may be 
suitably arranged in different portions of the housing 
10a (Figure 1) for providing different vibrations on the 
5 different portions of the user of the mobile phone 

(Figure 1) . 

For example, the electromechanical actuator 70 shown 
in Figure 9 may be suitably arranged as follows: 

(1) in a lower portion of a housing 10a of the 
10 mobile phone 10 for providing vibration on the 

user's fingers, 

;.: (2) in a lower or intermediate portion of a 

\° housing 10a of the mobile phone 10 for providing 

vibration on the user's facial skin, 
15 (3) in a lower or intermediate portion of a 

housing 10a of the mobile phone 10 for providing 
vibration on the user's wrist, or 

(4) in an intermediate portion of a housing 10a 
of the mobile phone 10 for providing vibration on 
20 the user's cheek. 

The scope of the invention is not intended to be limited 
to the arrangement of the vibrotactile actuator 42 
(Figure 2) in the housing 10a of the mobile phone 10. 
Moreover, the vibrotactile actuator 42 (Figure 2) 
25 may also be an acoustic actuator 80 shown in Figure 10 

suitably sized for fitting into a user's ear. 
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Figure 11: Telecommunications Network 100 
Figure 11 shows another embodiment of the present 
invention, wherein the telecommunications system 
generally indicated as 100 comprises an audio-to- 
5 vibrotactile converter 102. Consistent with that 

discussed above, the audio- to- vibrotactile converter 102 
may be implemented in hardware or software, and the scope 
of the invention is not intended to be limited to any 
particular implementation thereof. 

10 In operation, the audio- to-vibrotactile converter 

102 responds to an incoming speech signal from a 
called/calling party 111, for providing the 
telecommunications signal in the form of an audio-to- 
vibrotactile converter signal containing information 

15 about a vibration modulation of the incoming speech from 

the called/calling party 111. 

In this embodiment, the mobile phone 110 has an 
audio-modulated vibrotactile module 13 0 that only 
includes a vibrotactile actuator 142. The vibrotactile 

2 0 actuator 142 responds to the audio- to-vibrotactile 

converter signal from the audio- to-vibrotactile converter 
102, for providing the audio-modulated vibrotactile 
module force in the form of a vibrotactile actuator 
force . 

25 The telecommunications network 100 may be a 

separately-priced vibrotactile service network. 
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The audio-to-vibrotactile converter 102 may include 
a personalized hearing parameters module 15 0 shown in 
Figure 12 for adjusting speech processing so a user of 
the mobile phone can have personalized hearing 
5 parameters. The personalized hearing parameters can 

either be selected by a trial-and-error basis, preset 
values or personalized values given by a user's 
physician, for example. 



Advantages of the Invention 
10 Some advantages of the invention include the 

following : 

1) New hearing aid devices can be made available. 

2) The user's ears will not be the only channel to 
distribute speech for listening. 

15 3) People having significant hearing loss can use 

mobile phones easier than before (if previously at all 
possible) . The invention will also help hearing in a 
noisy environment where there is typically reduced 
hearing capability. 

2 0 5) The invention may be implemented with only minor 

hardware/software modification, which is primarily 
related to use of the mechanical vibration component. 

6) The invention has potential for integration with 
other audio enhancement features, for example, such as 

25 with active background noise reduction by shifting pitch 
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as related to background noise level to assist hearing 
with all other users as well. 

7) If a hearing loss will prevent someone from 
having conversations, they still are able to communicate 

5 with other people with the help of a mobile phone having 

this feature. This feature has the potential even to 
turn a phone into a hearing aid, leading to frequent and 
required every day use of the device . 

8) If a network-based solution is used, this feature 
10 will work with older phones as well and more processing 

power can be used. Also, a separately priced network 
service can be provided to a specific customer. 

9) Finally, some improvement of hearing capability 
can also be expected in a noisy environment with normal 

15 hearing ability as well. 

Scope of the Invention 
Accordingly, the invention comprises the features of 
construction, combination of elements, and arrangement of 
parts which will be exemplified in the construction 
2 0 hereinafter set forth. 

It will thus be seen that the objects set forth 
above, and those made apparent from the preceding 
description, are efficiently attained and, since certain 
changes may be made in the above construction without 
25 departing from the scope of the invention, it is intended 

that all matter contained in the above description or 
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shown in the accompanying drawing shall be interpreted as 
illustrative and not in a limiting sense. 
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